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Executive Summary 


Persistently low-performing schools have been repeatedly targeted for comprehensive reform for more 
than two decades, usually with poor results. These efforts have suffered, however, because they were 
often poorly implemented or insufficiently grounded in rigorous research. 


The Every Student Succeeds Act (ESSA), enacted in 2015, provides a framework for reversing this track 
record of failure through the greater adoption of proven, evidence-based practices and programs.? 
ESSA’s framework is not perfect, but if it is strengthened it could produce better results for the following 
reasons: 


e The Number of Evidence-based Programs and Practices, including Whole School Reform 
Models, Has Grown: Previous attempts to reform or rapidly turn around low-performing Title | 
schools have often foundered because they were insufficiently evidence-based, poorly- 
implemented, or both. However, far more evidence exists today than during the height of these 
previous turnaround efforts. These evidence-based programs and practices, including several 
whole school reform models, provide a pathway for proven, evidence-based change. 


e Although They Are Relatively Weak, ESSA Includes Evidence Provisions that Will 
Encourage Evidence-based School Improvement: Consistent with its overall goal of returning 
decision-making authority back to the states, ESSA eliminated the federal School Improvement 
Grants program and its mandated reform models. In their place, it provided schools and districts 
with more flexible funding and authority to devise their own reform plans, subject to a requirement 
that they include one or more interventions that are evidence-based, as defined by the law. 


These provisions, which are the focus of this paper, encourage schools and school districts to 
adopt more evidence-based strategies. By themselves, however, they may not be sufficient to 
induce widespread change. 


ESSA'‘s provisions only require comprehensive school reform plans to include a single 
intervention that meets the lowest ("promising") of the law’s top three evidence tiers. Moreover, 
the chosen interventions or strategies need not be new. Schools can avoid meaningful change by 
finding modest evidence that justifies a continuation of existing practices. Under ESSA, schools 
that wish to avoid evidence-based change can easily do so. Given the political and capacity 
challenges that confront any change in the status quo, many schools and school districts will 
likely do the minimum necessary to comply with the law. 


1 For more information, contact Patrick Lester, Director, Social Innovation Research Center, at (443) 822-4791 or 
patrick@socialinnovationcenter.org. 
2 ESSA (PL 114-95). Text available at: https://www.congress.gov/bill/114th-congress/senate-bill/1177/text. 
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Some States, Districts, and Schools May Exceed ESSA’s Minimum Requirements and 
Become Models for Other Jurisdictions and Schools: While ESSA’s evidence requirements 
for formula-funded programs are weak, both the history of previous reform efforts and the state 
ESSA plans submitted to the U.S. Department of Education for approval suggest that some 
jurisdictions will exceed the law’s minimum requirements. If they are successful, these evidence- 
based efforts may follow the pattern of previous education reforms that spread quickly across 
state lines. 


Policymakers could increase the likelihood of success if they adopted the following supportive changes: 


States and School Districts Should Adopt Stronger Evidence Standards than the Minimum 
Requirements Mandated by ESSA: While ESSA’s evidence requirements may be comparably 
weak, states and school districts can strengthen them by requiring plans for low-performing Title | 
schools identified for comprehensive improvement to include: (1) at least one new evidence- 
based intervention; (2) at least one intervention, new or current, that meets ESSA’s highest 
(“strong”) evidence standard; and/or (3) a proposal to test the effectiveness of at least one new 
and innovative practice, possibly as part of a research-practice partnership. 


More States Should Identify Evidence-based Practices and Providers with Strong Track 
Records: ESSA grants substantial freedom to states to determine which interventions and 
strategies meet its evidence definitions. Some states have indicated that they are developing, or 
will develop, lists of proven interventions. More should do so, while simultaneously considering 
certain needs such as ensuring that the lists are large enough and flexible enough to address 
gaps identified in school needs assessments. 


Given the central role that intermediaries can play in assisting with fidelity and implementation 
quality, states should also consider vetting developers, providers, and/or consultants. 


Federal Competitive Grants Should Be Used to Bolster Strong Evidence-based Efforts: 
Each year, the federal government provides over $2 billion in competitive grants for K-12 
education. The U.S. Department of Education has already adopted evidence requirements for 
these grants. They may be further strengthened by their competitive nature. If administered 
effectively, they could bolster evidence-based efforts at the state and local levels and further build 
the existing evidence base. States should also make use of federal programs such as the 
Regional Educational Laboratories and Comprehensive Centers to support their work in these 
areas. 


States Should Integrate Evidence More Thoroughly Into their Accountability Systems, 
Grants, and Technical Assistance: To help overcome forces that may resist changes to the 
status quo, states should better incorporate evidence into their accountability systems, make 
grants competitive or contingent upon the use of such evidence, and provide targeted assistance 
that is adapted to the needs of these schools, which typically experience high staff turnover, 
challenging learning environments, and severe capacity constraints. 


The Case for Evidence-Based Reform 


One of the most consistent targets of education reform over the past several decades has been the 
nation’s lowest-performing schools. Sometimes referred to as “drop-out factories” when considered in a 
high school context, these schools are often dysfunctional in every important way — including poor 
leadership, low-performing teachers, and disruptive school environments.? 
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Associated Press, "One in Ten US High Schools is a Dropout Factory," October 29, 2007. Available at: 
http://www.nbcnews.com/id/21531704/ns/us_news-education/t/us-high-schools-dropout-factory/; Mass Insight, "The Turnaround 
Challenge," September 2007, pp. 10, 24-29, 50-51, 96.-99 . Available at: http://www.massinsight.org/wp- 
content/uploads/sites/2/2015/11/TheTurnaroundChallenge SupplementalReport.pdf 
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Improving such schools, some argue, requires aggressive whole school transformation.* However, after 
nearly two decades of work — including both rapid school “turnarounds” that featured substantial staff and 
culture changes and slower, more incremental, comprehensive improvements — these efforts have failed 
to yield substantial national gains in student achievement.® 


Despite this disappointing overall track record, however, there have been pockets of success at the state 
and local levels. These isolated examples suggest that a new strategy rooted in proven, evidence-based 
practices could succeed where previous efforts have failed. 


School Turnarounds: A Track Record of Failure 


Federal aid for schools serving disproportionately low-income student populations dates back to the 
Elementary and Secondary Education Act (ESEA) of 1965, which provided billions of dollars in assistance 
through Title |, a portion of which was devoted to low-performing schools. ESEA’s enactment was part of 
a larger policy focus on increased equity, which included civil rights legislation and the Johnson 
administration’s broader War on Poverty.’ While greater equity in school funding has been important, 
however, it has not been sufficient to produce better student outcomes. 


Starting in the 1980s, following the publication of A Nation at Risk — a comprehensive report that 
recommended strengthening school curricula and establishing measurable standards for schools, 
textbooks, instructional materials, and standardized tests — the nation’s governors and a succession of 
presidents began to take a heightened interest in the performance of Title | schools.® 


In 1994, Congress began attaching accountability requirements to Title | funds, but states were slow to 
implement them.’ That changed in 2002 with the enactment of No Child Left Behind (NCLB), which 
strengthened the earlier accountability provisions and authorized additional, targeted aid through a newly- 
created School Improvement Grants (SIG) program.1° 


In 2009, President Obama strengthened the federal commitment to this framework. Under federal 
stimulus legislation enacted in the first days of his administration, the SIG program was granted a one- 
time boost of $3 billion, with an additional $500 million per year in subsequent annual appropriations. "! 
These grants provided the affected low-performing schools with up to $2 million per year in assistance 
over three years and mandated the adoption of the following four school reform strategies: '2 


4 Robert Balfanz, et al., “Building a Grad Nation," November 2010. Available at: https://www.edweek.org/media/14grad1.pdf. 
Annual updates are available at: http://www.americaspromise.org/building-grad-nation-report 
5 Lisa Dragoset, et al., "School Improvement Grants: Implementation and Effectiveness," Institute of Education Sciences, January 


2017. Available at: https://ies.ed.gov/ncee/pubs/20174013/ 
& Congressional Research Service, "History of the ESEA Title I-A Formulas," July 17, 2017. Available at: 


https://www.everycrsreport.com/reports/R44898.html 

7 Jack Jennings, "Reflections on a Half-century of School Reform: Why Have We Fallen Short and Where Do We Go from Here?", 
Center on Education Policy, January 12, 2012, pp. 2-3. Available at: http://www.cep- 
dc.org/displayDocument.cfm?DocumentID=392 

8 National Commission on Excellence in Education, "A Nation at Risk," April 1983. Available at: 
https://www2.ed.gov/pubs/NatAtRisk/index.html; Alyson Klein, "Historic Summit Fueled Push for K-12 Standards," Education 
Week, September 23, 2014. Available at: https://www.edweek.org/ew/articles/2014/09/24/05summit.h34.html 

® Congressional Research Service, "Accountability Issues and Reauthorization of the Elementary and Secondary Education Act," 
February 7, 2011, p. 6. Available at: https://www.everycrsreport.com/reports/R41533.html; GAO, "Stronger Accountability 
Needed for Performance of Disadvantaged Students," June 1, 2000. Available at: httos://www.gao.gov/products/HEHS-00-89 

10 Congressional Research Service, "Accountability Issues and Reauthorization of the Elementary and Secondary Education Act," 
February 7, 2011. Available at: https://www.everycrsreport.com/reports/R41533.html; Although SIG was authorized in NCLB, it 
was not funded until 2007. Jessica Quillin, “Snapshot of SIG: A Look at Four States’ Approaches to School Turnaround.” Center 
for American Progress, 2011. Available at https://cdn.americanprogress.org/wp- 
content/uploads/issues/2011/06/pdf/sig_report.pdf. 

1 The stimulus-related funding was included in the American Recovery and Reinvestment Act of 2009 (P.L 111-5). A review of the 
history of SIG program in the Obama administration’s first term is in: GAO, "Early Implementation Under Way, but Reforms 
Affected by Short Time Frames," July 25, 2011. Available at: https://www.gao.gov/products/GAO-11-741. 

12 The four SIG models are discussed in greater detail in: Institute of Education Sciences, “State Capacity to Support School 
Turnaround,” May 2015, pp.13-14. Available at: https://ies.ed.gov/ncee/pubs/20154012/pdf/20154012.pdf; Many of the elements 
of these four models also existed under NCLB, including school choice provisions and various corrective action requirements. 
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1. Closure: Under this option, schools would be shut down and their students transferred to other, 
higher-performing schools. '? 


2. Restart: Under this option, schools would be closed and restarted as charter schools. 


3. Turnaround: Under this option, the school principal and 50 percent of the school’s teachers 
would be replaced. The school would also be required to implement certain reforms. '4 


4. Transformation: Under this option, the principal would be replaced and the school would be 
required to implement specified reforms.'> This option did not require teachers to be replaced, 
although the reforms included teacher evaluation and accountability provisions. 


Starting in 2011, the Obama administration began issuing waivers to NCLB’s accountability provisions 
that deepened its commitment to turnaround strategies.'° These included regulations that directed states 
to adopt seven core turnaround principles, including those covering school leadership, teaching, 
additional learning time, instructional practices, data use, school climate, and family engagement.'” 


These investments usually produced poor results. Early data from the Department of Education showed 
modest academic gains.'® These results were later echoed by a national evaluation of the SIG program, 
which suggested that it produced modest changes in school-level practices and little overall impact on 
student achievement.'9 


After years of significant disruption coupled with modest results, the political tide turned. The Every 
Student Succeeds Act (ESSA), enacted in 2015, substantially rolled back many of NCLB’s accountability 
requirements, shifting greater responsibility to the states.2° It also eliminated the SIG program and 
replaced it with a new, more flexible funding stream for states and districts under Title | worth 
approximately $1 billion per year.2' The new law directed districts to devise their own turnaround 
strategies for these schools, subject to a minimal requirement that they include one or more “evidence- 
based” interventions that meet one of the law’s top three evidence standards.2 


See Congressional Research Service, "Accountability Issues and Reauthorization of the Elementary and Secondary Education 
Act," February 7, 2011, pp, 10-17. Available at: https://www.everycrsreport.com/reports/R41533.html; 

13 Chunping Han, et al., "School Closure Study 2017," Center for Research on Education Outcomes, August 24, 2017. Available at: 
http://credo.stanford.edu/closure-virtual-control-records 

14 For a list of reforms, see Institute of Education Sciences, "Are Low-performing Schools Adopting Practices Promoted by School 
Improvement Grants?", October 2014, p. 7. Available at: https://ies.ed.gov/ncee/pubs/2015400 1/pdf/20154001.pdf 

18 Ibid. 

16 CRS, "Reauthorization of the Elementary and Secondary Education Act: Highlights of the Every Student Succeeds Act," 


December 4, 2017, pp. 1-2. Available at: https://www.everycrsreport.com/reports/R44297.html 

17 U.S. Department of Education, "ESEA Flexibility 2012,” 2012. Available at http://www2.ed.gov/policy/eseaflex/approved- 
requests/flexrequest.doc. 

18 Alyson Klein, "New Data Paints Mixed Picture of Federal Turnaround Program," Education Week, December 1, 2015. Available 
at: https://www.edweek.org/ew/articles/2015/12/02/new-data-paints-mixed-picture-of-federal.html 

19 Lisa Dragoset, et al., "School Improvement Grants: Implementation and Effectiveness," Institute of Education Sciences, January 
2017. Available at: https://ies.ed.gov/ncee/pubs/20174013/; Lisa Dragoset, et al., "Usage of Practices Promoted by School 
Improvement Grants," Mathematica Policy Research, September 29, 2015. Available at: https://www.mathematica-mpr.com/our- 
publications-and-findings/publications/usage-of-practices-promoted-by-school-improvement-grants 

20 Aside-by-side comparison of ESSA and NCLB, including their treatment of accountability provisions, is in: CRS, 
"Reauthorization of the Elementary and Secondary Education Act: Highlights of the Every Student Succeeds Act," December 4, 
2017, pp. CRS-4 to CRS-11. Available at: https://www.everycrsreport.com/reports/R44297.html 

21 ESSA (PL 114-95). Text available at: httos://www.congress.gov/bill/114th-congress/senate-bill/1177/text. The new 7 percent set 
aside is in § 1003(a)(1). 

22 This process is described in various parts of the law. ESSA’s provisions governing state accountability systems are in § 1111(c). 
Provisions for designating schools in need of comprehensive improvement are in § 1111(c)(4)(D)(i). These schools are eligible 
for Title | school improvement funds under § 1003, which replaced the School Improvement Grants program. Planning 
requirements for designated schools are described in §§ 1111(d)(1)(B) and 1111(d)(2)(B). ESSA’s evidence definition can be 
found in § 8101(21)(A). The requirement to use one of the top three evidence tiers is in § 8101(21)(B). ESSA also has a lower 
tier, “demonstrates a rationale,” but this tier does not apply to Title | school improvement funds (see Appendix A). 
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Evidence as a Determinant of Success 


Looking back, the SIG program is often viewed as a failure, including by the Trump administration, which 
cited the program’s final evaluation as proof that the Obama administration’s school reform efforts were a 
waste of resources.22 Some conservatives went further, calling the program’s poor results predictable.24 


Were they? Some proponents of stronger reform said SIG failed because it was insufficiently ambitious.25 
School turnarounds are “difficult to impossible” to implement, they said, because of “meddling from 
teacher unions, the school board, or the central office.” 2° They claimed that the two most aggressive 
options — school closure and charter conversion — were the most likely to succeed, but rarely tried. 2” 


Opponents argued the opposite — that SIG failed because it was too aggressive. School closures were 
unnecessarily disruptive, they said, and the evidence of their effectiveness was weak.28 Even the least 
disruptive options, which required replacing principals (as well as teachers under the turnaround option), 
presented enormous logistical challenges.22 Some school districts merely rotated principals between 
their schools, for example, and subsequent studies found that replacing principals produced no gains in 
student achievement.°° 


Despite the competing claims of ardent supporters and opponents, however, a review of the SIG 
program’s history suggests a more nuanced answer. More than 90 percent of affected schools adopted 
one of the two least aggressive programs models — turnaround and transformation.*' These two models 
promoted the use of school practices that did not seem to substantially improve student outcomes.°? 


There were several reasons for this. First, the practices were insufficiently evidence-based.*° This was 
partly because few rapid-turnaround practices were backed by solid evidence at the time the program 
was created, something that was also true for turnarounds efforts in the public sector more broadly.%4 


23 Alyson Klein, "Betsy DeVos: 'Education Establishment’ Has Blocked Efforts to Fix Schools," Education Week, February 23, 
2017. Available at: http://blogs.edweek.org/edweek/campaign-k-12/2017/02/devos_cpac_hold.html 

Andy Smarick, "The $7 Billion School Improvement Grant Program: Greatest Failure in the History of the U.S. Department of 
Education?", January 1, 2017. Available at: http://educationnext.org/the-7-billion-school-improvement-grant-program-greatest- 
failure-in-the-history-of-the-u-s-department-of-education/ 

Tiffany D. Miller and Catherine Brown, "Dramatic Action, Dramatic Improvement: The Research on School Turnaround," Cener 
for American Progress, March 31, 2015. Available at: https://www.americanprogress.org/issues/education-k- 
12/reports/2015/03/31/110142/dramatic-action-dramatic-improvement/ 

Andy Smarick, "The Turnaround Fallacy," Education Next, Winter 2010. Available at: http://educationnext.org/the-turnaround- 
fallacy/ 

Ibid.; See also Howard Bloom and Rebecca Unterman, "Sustained Positive Effects on Graduation Rates Produced by New York 
City’s Small Public High Schools of Choice," MDRC, January 2012. Available at: https://www.mdrc.org/publication/sustained- 
positive-effects-graduation-rates-produced-new-york-city-s-small-public-high; GAO, "School Improvement Grants: Early 
Implementation Under Way, but Reforms Affected by Short Time Frames," July 25, 2011, p. 13. Available at: 
https://www.gao.gov/products/GAO-11-741 

Tina Trujillo, "Review of Dramatic Action, Dramatic Improvement," National Education Policy Center, May 2015. Available at: 
http://www.greatlakescenter.org/docs/Think_ Twice/TT-Trujillo-CAP-Turnarounds.pdf 

Center on Education Policy, "Schools with Federal Improvement Grants Face Challenges in Replacing Principals and Teachers," 
July 2012. Available at: https://www.cep-dc.org/displayDocument.cfm?DocumentID=406 

Rebecca Herman, et al.,, "School Leadership Interventions Under the Every Student Succeeds Act: Evidence Review," RAND, 
December 2017, p. 21. Available at: https://www.rand.org/pubs/research_reports/RR1550-3.html; Robert Slavin, "Transforming 
Transformation (and Turning Around Turnaround)," February 2, 2017. Available at: 
https://www.huffingtonpost.com/entry/5893b09be4b02bbb1816b8aa 

GAO, "School Improvement Grants: Early Implementation Under Way, but Reforms Affected by Short Time Frames," July 25, 
2011, p. 13. Available at: https://www.gao.gov/products/GAO-11-741 

52 Alyson Klein, "New SIG Data Serve Up Same Old Conclusion: Mixed Results," Education Week, November 12, 2015. Available 
at: http://blogs.edweek.org/edweek/campaign-k-12/2015/11/new_sig data_serves up same _ol.html; U.S. Department of 
Education, "School Improvement Grants National Summary: School Year 2012-13," November 12, 2015. Available at: 
https://www2.ed.gov/programs/sif/signationalsum09292015.pdf 

Patrick Lester, "School Improvement Grants Program’s Failure Points to Evidence-based Policy as an Answer," Social 
Innovation Research Center, February 4, 2017. Available at: http://www.socialinnovationcenter.org/?p=2543; Institute for 
Education Sciences, "Turning Around Chronically Low-performing Schools," May 2008. Available at: 
https://ies.ed.gov/ncee/wwc/Docs/PracticeGuide/Turnaround pg 04181.pdf 

Hamid Ravaghi, "Organizational Failure and Turnaround in Public Sector Organizations: A Systematic Review of the Evidence," 
December 11, 2017. Available at: http://miiri.iums.ac.ir/article-1-3392-en.pdf; George A. Boyne, "Strategies for Public Service 
Turnaround: Lessons from the Private Sector?", May 2006. Available at: 
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According to the SIG program’s final national evaluation: 


Though research on SIG is limited, a large body of literature examines the effectiveness of the 
school improvement practices promoted by SIG and school turnaround more broadly. Overall, 
this literature provides mixed evidence on whether these practices improve student outcomes.%5 


Second, the SIG program did little to increase the use of such practices.3° The national evaluation found 
that while SIG schools adopted more practices, the increase was small and statistically insignificant when 
compared to other schools, many of which had also been adopting these practices because of other 
federal programs like the Obama administration’s Race to the Top initiative. °” 


Finally, when schools implemented the practices, they often did so poorly. Low-performing schools 
commonly face significant staff turnover and severe capacity constraints, which hinder implementation.°8 
Some schools relied on a “kitchen sink” approach that attempted to apply many different reform strategies 
at once, with poor coordination and support.°9 


In short, the turnaround practices that SIG promoted were backed by mixed evidence, not implemented in 
significantly greater numbers, and often poorly implemented. In this light, the program’s negligible impact 
on student outcomes overall is unsurprising. 


Despite these overall challenges, however, there were also pockets of success.*° Some states like 
Massachusetts, Ohio, and California appeared to produce better results.*1 Some cities like New York and 
San Francisco also appeared to outperform their peers.*? Analyses cited significant differences in the 
aggressiveness with which reforms were implemented locally, which may have affected their results.** 
Others pointed to substantial differences in state laws and capacities, which may have contributed to 
variations in effectiveness .*4 


http://journals.sagepub.com/doi/abs/10.1177/0095399705286004?journalCode=aasb; Grover "Russ" Whitehurst, "Relaying on 
Evidence," December 2013. Available at: 


https://www.hoover.org/sites/default/files/research/docs/finnsousa_whatliesahead_final_ch10.pdf 

35 Lisa Dragoset, et al., "School Improvement Grants: Implementation and Effectiveness," Institute of Education Sciences, January 
2017. Available at: https://ies.ed.gov/ncee/pubs/20174013/ 

36 Ibid. 

3” The report indicated that SIG schools adopted an extra 2.5 practices out of a possible 35 when compared to non-SIG schools. 
See Lisa Dragoset, et al., "School Improvement Grants: Implementation and Effectiveness," Institute of Education Sciences, 
January 2017, p. ES-2, ES-13. Available at: https://ies.ed.gov/ncee/pubs/20174013/ 

38 Kerstin Carlson Le Floch, et al, "Case Studies of Schools Receiving School Improvement Grants," April 2016. Available at: 
https://eric.ed.gov/?id=ED565615 

39 Sarah Yatsko, et al., "Tinkering Toward Transformation," Center on Reinventing Public Education, March 2012. Available at: 
https://files.eric.ed.gov/fulltext/ED532630.pdf 

40 Matt Barnum, "Betsy DeVos Called Obama’s School Turnaround Program a Failure, But New Research Shows It Worked — ina 
Few Places," Chalkbeat, May 9, 2017. Available at: https://www.chalkbeat.org/posts/us/2017/05/09/betsy-devos-called-obamas- 


school-turnaround-program-a-failure-but-new-research-shows-it-worked-in-a-few-places/ 


41 Christina LiCalsi, "Evaluation of Massachusetts Office of District and School Turnaround Assistance to Commissioner’s Districts 
and Schools," AIR, June 2015. Available at: http://www.air.org/sites/default/files/downloads/report/15-2687_SRG_Impact- 
Report_ed FINAL.pdf; Ohio Department of Education, "Ohio School Turnaround Interventions: Impacts on Achievement, 
Attainment, and Administration," October 27, 2016. Available at: http://oerc.osu.edu/index.php/ohio-school-turnaround- 
interventions-impacts-on-achievement-attainment-and-administration/; Susanna Loeb, "Continued Support for Improving the 
Lowest-performing Schools," Brookings Institution, February 9, 2017. Available at: 
https://www.brookings.edu/research/continued-support-for-improving-the-lowest-performing-schools/; Daniel Player and 
Veronica Katz, "School Improvement in Ohio and Missouri: An Evaluation of the School Turnaround Specialist Program,” May 
2013. Available at: http://curry.virginia.edu/uploads/resourceLibrary/10_ Player SchoolTurnaround.pdf 

42 Jane L. David and Joan E. Talbert, “Turning Around a High-Poverty District: Learning from Sanger” (San Francisco: S.H. Cowell 
Foundation, 2013). Available at http://shcowell.org/wp-content/uploads/2015/12/Learning-From-Sanger.pdf; MDRC, "Sustained 
Positive Effects on Graduation Rates Produced by New York City’s Small Public High Schools of Choice," January 2012. 
Available at: httos://www.mdrc.org/publication/sustained-positive-effects-graduation-rates-produced-new-york-city-s-small-public- 
high; Gov Innovator, “Insights from a Leading University-School District Partnership," January 11, 2018. Available at: 
http://govinnovator.com/laura_wentworth/ 

43 Sarah Yatsko, et al., "Tinkering Toward Transformation," Center on Reinventing Public Education, March 2012. Available at: 
https://files.eric.ed.gov/fulltext/ED532630.pdf;Tiffany D. Miller and Catherine Brown, "Dramatic Action, Dramatic Improvement: 
The Research on School Turnaround," Center for American Progress, March 31, 2015. Available at: 


https://www.americanprogress.org/issues/education-k-12/reports/2015/03/31/110142/dramatic-action-dramatic-improvement/ 
Institute of Education Sciences, “State Capacity to Support School Turnaround,” May 2015, pp.13-14. Available at: 


44 
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Rigorous studies also identified some privately-developed whole school reform models with established 
track records of success, such as Success for All, the Institute for Student Achievement, and the Positive 
Action System. In 2014, Congress enacted legislation allowing schools in the SIG program to adopt these 
evidence-based whole school turnaround models, but the change came late in the program’s life.45 Few 
schools adopted them before the SIG program was eliminated by ESSA.46 


Nevertheless, these successes at the state, local, and provider levels suggest a different conclusion 
about the SIG program and turnaround efforts in general. When such initiatives failed, they may have 
done so not because school transformation was unworkable, but because the chosen strategies were 
insufficiently evidence-based, poorly implemented, or both. By contrast, states, schools, and private 
model developers that overcame those challenges seemed more likely to succeed. 


Evidence Building 


Throughout most of the 1990s and 2000s, when work on school turnarounds was either beginning or just 
becoming well-established, there was little rigorous evidence to demonstrate its effectiveness. A 2008 
review sponsored by the Institute of Education Sciences (IES) at the U.S. Department of Education. found 
no studies of rapid turnaround practices (as opposed to longer-term, comprehensive reforms) that met 
What Works Clearinghouse standards for experimental or quasi-experimental research.*” Despite the 
relative paucity of evidence that existed at the time, however, there has been significant progress since 
then. 


Individual Practices 


School reforms can target any number of practices, but the turnaround literature has highlighted a small 
handful of topics as particularly important: principals, teachers, instruction, and school climate.*® 


e Principals: According to one comprehensive review, school leadership (broadly defined, but 
including principals) is one of the most important school-based drivers of student achievement, 
second only to teachers.*9 Effective school principals hire higher-quality teachers, have teachers 
who improve faster, and experience lower teacher turnover.°° An IES review of studies of the 
School Improvement Grants program found an association between those schools that improved 


https://ies.ed.gov/ncee/pubs/20154012/pdf/20154012.pdf; Institute of Education Sciences, "State Policies for Intervening in 
Chronically Low-performing Schools: A 50-state Scan," June 2016. Available at: 
https://ies.ed.gov/ncee/edlabs/regions/central/pdf/REL_2016131.pdf; Center on Reinventing Public Education, "The Capacity 
Challenge," December 2013. Available at: https://www.crpe.org/sites/default/files/oub_capacity%20challenge dec13_ 0.pdf ; 
GAO, "Early Implementation Under Way, but Reforms Affected by Short Time Frames," July 25, 2011, pp. 22-27. 
Consolidated Appropriations Act, 2014 (Public Law No: 113-76); Alyson Klein, "SIG Program Gets Makeover in Newly Passed 
Budget," Education Week, January 28, 2014. Available at: https://www.edweek.org/ew/articles/2014/01/29/1 9budget- 
sig.h33.html; Implementing regulations for can be found at: https://www.federalregister.gov/documents/2014/09/08/2014- 
21185/proposed-requirements-school-improvement-grants-title-i-of-the-elementary-and-secondary-education 
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and strong leadership.*" 


A 2017 RAND literature review identified several evidence-based programs that could improve 
school leadership. These include the New Leaders Aspiring Principals Program (principal 
preparation), Texas Principal Excellence Program (principal preparation), National Institute for 
School Leadership Executive Development Program (professional learning), and principal 
autonomy (working conditions).52 


e Teachers: Teachers are the most important in-school driver of student achievement.°? Teacher 
quality can be affected through recruitment, retention, and replacement strategies.** It can also 
be affected by teacher evaluations coupled with targeted teacher professional development.® 
Comprehensive school improvement efforts under SIG featured each of these strategies. °6 


e Instructional Practices: Successful schools cultivate an environment of high expectations for 
academic achievement and use instructional practices that are standards-based, individualized, 
and rely on research-based teaching methods.°’ The What Works Clearinghouse has released 
practice guides on instruction in reading, writing, and mathematics.*® It has rated specific 
programs for literacy, math, science, English Language Learners, students with disabilities, and 
early childhood (pre-K) instruction.5° The independent site Evidence for ESSA has also rated 
specific math and reading programs like Reading Recovery and Number Rockets.®° 


e School Climate: Learning is easier in schools with safe, orderly school environments that are 
free of bullying and disciplinary problems.®' SIG schools frequently implemented programs 
intended to promote order and discipline, including adopting school uniforms, hiring additional 
staff to focus on student behavior management, or instituting specific interventions like Positive 
Behavior Interventions and Supports (PBIS).°* The What Works Clearinghouse has rated several 
related programs such as First Step to Success and Positive Action. 
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The importance of these factors has been repeatedly noted in the practitioner literature, usually in the 
form of general principles, but they are frequently implemented with little demonstrated impact.® 
Evidence-based programs and practices are usually more prescriptive than such principles, however, with 
strict fidelity requirements for core components such as staffing, activities, and materials. The number of 
evidence-based programs and practices that meet these criteria is growing, however. 


Ongoing funding for additional research on such interventions principally comes from IES, the Education 
Research and Innovation (EIR) program (formerly the Investing in Innovation program, or i3), and other 
federal competitive grant programs that include evidence-building requirements.® Some states also 
provide funding for research.® Philanthropy also plays a supporting role. 


Whole School Models 


Individual evidence-based practices may be important, but they may have only modest and incremental 
effects in a school that is not otherwise being subjected to substantial change. Analyses have suggested 
that school reform strategies that are more comprehensive and coherent are more likely to succeed. ®’ 


Today, there are at least six comprehensive reform strategies that are evidence-based, four of which have 
been independently reviewed and approved by the U.S. Department of Education as meeting evidence 
requirements for the SIG program.® These include: 


e Building Assets, Reducing Risks (BARR): The BARR program combines small study cohorts 
with professional development to increase academic performance by 9th graders.®? A randomized 
controlled trial (RCT) funded by the federal i3 program found that enrolled students earned more 
core credits, obtained better grades, experienced lower course failure, and earned higher test 
scores in reading and mathematics than students not enrolled in the program.”° The study has 
been independently reviewed and included in the What Works Clearinghouse.” 


e Diplomas Now: Diplomas Now is a comprehensive school reform program for middle schools 
and high schools that provides targeted interventions to students who exhibit “early warning 
indicators” of poor attendance, behavior, or academic performance. It is based on a partnership 
between Talent Development Secondary, City Year, and Communities In Schools.’? According to 
an interim, multi-site RCT evaluation, after two years it reduced the percentage of students 


64 Numerous guides describe principles for turning around low-performing schools. One early example is Mass Insight, "The 


Turnaround Challenge," September 2007. Available at: http://www.massinsight.org/wp- 
content/uploads/sites/2/2015/11/TheTurnaroundChallenge SupplementalReport.pdf 
Federal Register, "Department of Education: Definitions and Selection Criteria That Apply to Direct Grant Programs," July 31, 
2017. Available at: https://www.federalregister.gov/documents/2017/07/31/2017-15989/definitions-and-selection-criteria-that- 
apply-to-direct-grant-programs 
One example of state-based research on turnarounds can be found in Tennessee. See Erin O'Hara, "What 5 Years of Research 
Say About School Turnaround Efforts in Tennessee," Education Week, November 6, 2017. Available at: 
http://blogs.edweek.org/edweek/urban education reform/2017/11/school turnaround efforts in tennessee.html and Nate 
Schwartz, "Difficult Conversations: Learning from Tennessee's Turnaround Efforts," Education Week, November 9, 2017. 
Available at: 
http://blogs.edweek.org/edweek/urban_ education reform/2017/11/difficult_conversations learning from tennessees turnaroun 
d_efforts.html; A list of prominent state and local research-practice partnerships can be found at http://nnerpp.rice.edu/members/ 
William Corrin, “Reforming Underperforming High Schools” (New York: MDRC, 2013), available at 
http://www.mdrc.org/sites/default/files/High School Reform 030513 0.pdf; Council of Great City Schools, "School Improvement 
Grants: Progress Report from America’s Great City Schools," February 2015. Available at: 
https://www.cgcs.org/cms/lib/DC00001581/Centricity/Domain/87/SIG%20Report%202015.pdf 
68 U.S. Department of Education, "Approved Evidence-Based, Whole School Reform Models." Available at: 
https://www2.ed.gov/programs/sif/sigevidencebased/index.html; U.S. Department of Education, Final Requirements—School 
Improvement Grants—Title | of the Elementary and Secondary Education Act of 1965, Federal Register 80 (26) (2015): 7224— 
7251. Available at https://www.gpo.gov/fdsys/pkg/FR-2015-02-09/pdf/2015-02570.pdf 
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See: http://www.barrcenter.org/results 
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Turn Around Low-achieving Schools. Final Report," December 2016. Available at: https://ies.ed.gov/ncee/wwe/Study/132 
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exhibiting one or more early warning signs that a student will drop out, including poor behavior, 
low attendance, or poor academic performance.’”? 


e =Institute for Student Achievement (ISA): ISA is a high school redesign organization that has 
managed whole school reform efforts in Atlanta, Detroit, and Minneapolis.’ The model includes 
college prep, extended school day and year, parental involvement, and other practices.”°> A 2010 
evaluation found that it increased high school completion, achievement, and college 
preparation.’”¢ It is one of four programs that was approved by the Department of Education as an 
evidence-based whole school reform model. 


e New York City Small Schools Initiative: Small schools of choice are high schools with smaller 
student enrollments that feature personalized relationships and academic rigor. They have been 
found to improve graduation rates and increase college enrollment.’” They are one of four whole 
school programs approved by the Department of Education as evidence-based. 


e Positive Action: Positive Action is a scripted lessons program for pre-K to high school 
students.’ In a matched-pair cluster-randomized RCT it was found to improve academic 
achievement, absenteeism, and disciplinary outcomes.” It was one of four programs approved 
by the Department of Education as an evidence-based whole school reform model. 


e Success for All (SFA): SFA is a comprehensive school reform model for students from pre- 
kindergarten through eighth grade. It includes an extensive reading program, job-embedded 
professional development, and curriculum resources and strategies for addressing school-wide 
issues such as low attendance, parental involvement, school culture, family needs, and health 
issues.®° In separate studies it has been found to improve reading comprehension and produce 
positive effects on phonics. It was one of four programs approved by the Department of 
Education as an evidence-based whole school reform model. 


Evidence Use 


There may be more evidence-backed programs and practices today than during earlier efforts at reform, 
but the existence of such evidence does not guarantee its use. How can such interventions become more 
widely adopted? What impact, if any, will they have on low-performing Title | schools? 


This section begins with a review of ESSA’s evidence provisions. It then describes a general model for 
how evidence makes its way into practice, followed by a review of the specific infrastructures that exist at 
the federal, state, and local levels that could promote its use. 


The analysis concludes by suggesting that while ESSA’s evidence provisions may seem to be weak, 
these surface appearances may be deceiving. As in the past, there will likely be significant variation 
among states, districts, and schools in their adoption of evidence-based interventions, with some 
performing at much higher levels than others. If these early adopters succeed, however, history suggests 
that their success may be sufficient to spread their chosen strategies to other schools across the nation. 


MDRC, "Addressing Early Warning Indicators: Interim Impact Findings from the Investing in Innovation (i3) Evaluation of 

Diplomas Now," June 2016. Available at: https://www.mdrc.org/publication/addressing-early-warning-indicators 

74 See: https://www.studentachievement.org/ 

See: https://www2.ed.gov/programs/sif/sigevidencebased/isawsrnarrative.pdf 

76 See: https://www2.ed.gov/programs/sif/sigevidencebased/isaevidence.pdf 

77 MDRC, "New York City Small Schools of Choice Evaluation." Available at: https://www.mdrc.org/project/new-york-city-small- 
schools-choice-evaluation#overview 

78 See: https://www.positiveaction.net/ 

72 See: https://www2.ed.gov/programs/sif/sigevidencebased/positiveactionevidence.pdf 

See: http://www.successforall.org/ 
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ESSA’s Evidence Requirements 


At first glance, ESSA‘s evidence requirements do not seem to be a promising model for diffusing 
evidence-based programs and practices. Instead, they seem more in line with the politics that drove 
ESSA, which were a backlash against what some viewed as a period of federal overreach under NCLB, 
including under both the Bush and Obama administrations.®' 


Consistent with this overall direction, ESSA eliminated the SIG program, its four school reform models, 
and sent substantial decision-making authority over Title | schools back to the states.® In their place, it 
established a new approach that allowed districts and schools to adopt their own reform plans, subject to 
the requirement that they include one or more interventions that were evidence-based.®3 


These new evidence requirements could be viewed as weak for several reasons. First, the provisions are 
not new. NCLB also included requirements that its programs be rooted in “scientifically-based research.”® 
Second, ESSA only requires that low-performing Title | schools designated for comprehensive 
improvement have plans that include a single intervention that meets the lowest (“promising”) of the law’s 
top three evidence tiers.®5 Third, the selected evidence-based interventions would not need to be new. A 
school could minimally comply by finding relatively low “promising” evidence that justified strategies it was 
already pursuing. In short, ESSA‘s evidence provisions by themselves constitute barely a nudge for any 
district or school that does not wish to change. 


This apparent weakness may be deceiving, however. NCLB’s similar requirements for reforms backed by 
scientifically-based research were undercut by the narrowness of the definition and the general lack of 
such evidence that existed at the time.®° This is no longer true. While more research is needed, much 
more exists today than during most of the NCLB era (as described earlier in this paper). 


Moreover, while ESSA’s evidence provisions appear to have set a low bar, their potential strength 
becomes more apparent after considering the surrounding federal, state, and local infrastructures that 
remain largely intact after its adoption. ESSA may have pushed more control over education decisions to 
the states, but it did not eliminate the state accountability mechanisms, other capacities, or state and local 
political dynamics that supported, and continue to support, the reform of low-performing schools. 


In short, ESSA may have repealed some of NCLB’s most significant provisions, but NCLB’s legacy 
remains. ESSA’s evidence provisions may be weak, but other forces that incentivize the use of such 
evidence may be correspondingly strong, thereby overcoming some of ESSA’s weaknesses. 


Three Evidence Pathways 

How might ESSA promote the adoption of evidence-based programs and practices? A complete answer 
requires a more detailed review of existing federal, state, and local laws and capacities. Understanding 
these processes can be made simpler, however, by viewing them as promoting one or more of the 


following evidence pathways: 


e Top-Down Evidence Pathway: Under this pathway, a federal, state, or local government adopts 


81 Julie Hirschfeld Davis, "President Obama Signs Into Law a Rewrite of No Child Left Behind," The New York Times, December 
10, 2015. Available at: https://www.nytimes.com/2015/12/11/us/politics/president-obama-signs-into-law-a-rewrite-of-no-child-left- 
behind.html 

82 Aside-by-side comparison of ESSA and NCLB, including their treatment of accountability provisions, is in: CRS, 
"Reauthorization of the Elementary and Secondary Education Act: Highlights of the Every Student Succeeds Act," December 4, 
2017, pp. CRS-4 to CRS-11. Available at: https://www.everycrsreport.com/reports/R44297.html 

83 ESSA §§ 1111(d)(1)(B) and 1111(d)(2)(B) 

84 Sarah Sparks, "NCLB Rewrite Sets New Path on School Research," Education Week, January 5, 2016. Available at: 
https://www.edweek.org/ew/articles/2016/01/06/nclb-rewrite-sets-new-path-on-school.html 

85 ESSA § 1111(d)(1)(B)(ii) and § 8101(21) 

86 Debra Viadero, "Research Effort Aims to Bury 'Nothing Works' Image," Education Week, December 15, 2008. Available at: 
https://www.edweek.org/ew/articles/2008/12/15/16whatworks.h28.html 
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policies (legislation, budgets, regulations) that induce policy changes by lower levels of 
government and practice changes in schools.8’ ESSA is an example. Such efforts can 
themselves be informed by evidence and by the advocacy efforts of intermediaries.®° This 
pathway is featured in a profile of turnaround efforts in Massachusetts later in this paper. 


e Bottom-Up Evidence Pathway: This pathway relies on high-capacity school districts, schools, or 
nonprofit partners that voluntarily adopt existing evidence-based interventions on their own.8° The 
pathway often relies on networks to share information between researchers and practitioners, 
such as professional associations and informal connections among trusted peers.°° The bottom- 
up pathway also includes organizations that develop and test new innovations of their own.*' 

This pathway is featured in a profile of San Francisco’s turnaround work later in this paper. 


Because it depends on actions originating at the local level, this pathway could be viewed as 
independent of policies adopted at higher levels of government. In practice, this is rarely true. At a 
minimum, such work usually depends, at least in part, on public funding for the selected 
practices.°2 Funding for research on K-12 education issues also predominantly comes from 
public sources.? 


e Provider-based Evidence Pathway: This pathway is more indirect. Under this approach, 
governments (and sometimes philanthropic organizations) provide funding to evidence-based 
program providers, which then reach out to and work with schools and school districts on a 
voluntary basis. Examples include the scaling efforts that took place in the federal Education 
Innovation and Research (EIR) program.® This pathway is featured in the review of the Success 
for All model later in this paper. 


Evidence use — including the evidence promoted by ESSA's evidence definitions — relies on all three of 
these pathways. Understanding how evidence follows these paths, however, requires a better 
understanding of the varying roles of federal, state, and local actors. 


The Federal Role 


The federal government provides evidence-related assistance for Title | schools in several ways, 
including: 


= Funding for Research: While there is some disagreement among policymakers and advocates 
over the appropriate role of the federal government in education in general, there is more 
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education-policy; 
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http://ncrpp.org/assets/documents/NCRPP_Technical-Report-1.pdf 

William Penneul, et al., "How School and District Leaders Access and Use Research," University Libraries Open Access Fund 
Supported Publications, April 19, 2017. Available at: http://scholar.colorado.edu/libr_oafund/32 

New innovations in education can sometimes be identified by examining variations in outcomes. See Improvement Science 
Research Network, "What is Improvement Science." Available at: http://isrn.net/about/improvement_science.asp 

Public funding — federal, state, and local — is the primary sources of K-12 education spending. For example, see: Richard Lee 
Colvin, "The New Philanthropists," Education Next, 2005. Available at: http://educationnext.org/thenewphilanthropists/ 
According to one analysis, the 15 largest foundation funders of K-12 education donated $52 million for education research in 
2013. IES funding on education research in FY 2013 was $190 million. See: Jay P. Greene, "Buckets into Another Sea," pp. 24- 
25 in Frederick Hess and Jeffrey Henig, The New Education Philanthropy, Harvard Education Press, December 1, 2015; U.S. 
Department of Education, "Institute of Education Sciences Fiscal Year 2014 Budget Request," April 2013, p. X-12. Available at: 
https://www2.ed.gov/about/overview/budget/budget14/justifications/x-ies.pdf 
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agreement about its role as a principal funder of research.% Most funding for K-12 education 
research comes from the Institute of Education Sciences (IES). Additional funding comes through 
the Education Innovation and Research (EIR) program.°* Many other federal competitive grants 
also include evaluation incentives or requirements.9” 


= Evidence Reviews: To be useful, research findings must be available to the public and, given the 
potential methodological problems that confront such research, ideally reviewed by a credible and 
independent third party.°8 IES performs both functions. It makes studies available through its 
Education Resources Information Center (ERIC), an online library of studies and other 
information.% It also operates the What Works Clearinghouse (WWC), which reviews and rates 
the studies of specific programs.1°° 


The WWC has been the subject of some criticism, however, including both the timeliness and 
methodology of its reviews.'°' These problems are not unique to the WWC. Such clearinghouses 
often struggle to balance rigor and usability.'!°* However, these issues may have reduced its 
usefulness. One study on evidence use by schools and districts suggests that the WWC is 
utilized less frequently than other sources of information, such as professional associations, 
trusted colleagues, and even newspapers and magazines. '°° 


=» Formula Grant Requirements: The existence of evidence is not sufficient to guarantee its use. 
ESSA has bridged this gap by defining “evidence-based” (see Appendix A) and attaching this 
definition to the requirements of several formula and competitive grants (see Appendix B). The 
influence of these definitions differs for these two types of grants, however. 


Perhaps owing to the underlying politics of ESSA, which reflected a reaction against perceived 
federal excesses under NCLB, the Department of Education has taken a lighter approach for 
formula grants. For these grants — including Title | funds for schools and Title II funds for teacher 
and principal-related programs — the Department of Education has issued only non-binding, non- 
regulatory guidance for implementing ESSA’s evidence provisions.'% 


The department has maintained some oversight responsibilities through its authority to approve 
state plans, but it does not seem to have used this authority to influence state decisions on the 
use of evidence. '%© 


95 


96 


97 


98 


99 


100 
101 


102 


103 


104 


105 


Michael A. Gottfried, et al., "Federal and State Roles and Capacity for Improving Schools," RAND, 2011. 
https://www.rand.org/pubs/technical_reports/TR989.html; GAO, "Education Research: Further Improvements Needed to Ensure 
Relevance and Assess Dissemination Efforts," December 5, 2013. Available at: http://www.gao.gov/products/GAO-14-8 

Patrick Lester, "Investing in Innovation (i3): Strong Start on Evaluation and Scale, But Greater Focus Needed on Innovation," 
Social Innovation Research Center, January 19, 2017. Available at: http://www.socialinnovationcenter.org/?p=2482 

Michele McNeil, "Evidence Matters: U.S. Education Department Finalizes EDGAR Rules," Education Week, August 14, 2013. 
Available at: http://blogs.edweek.org/edweek/campaign-k-12/2013/08/evidence_matters us education .html 

Such methodological problems threaten the validity of the evidence-based model. See SIRC, "Addressing the Research 
“Replication Crisis”: Evidence-based Policy’s Hidden Vulnerability," January 19, 2018. Available at: 
http://www.socialinnovationcenter.org/archives/2906 

See: https://eric.ed.gov/ 

See: https://ies.ed.gov/ncee/wwc/ 
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= Competitive Grant Requirements: The U.S. Department of Education has taken a stronger role 
on evidence in its competitive grants, which together amount to over $2 billion in assistance per 
year for K-12 education.'°° In 2017, the department issued regulations governing the use of 
evidence in these grants.1°” 


Competitive grants are a potentially powerful tool for influencing state, district, and school 
decision-making. The potential influence of these grants was amply demonstrated by the federal 
Race to the Top initiative.'°® Depending on how they are administered, these grants could 
provide incentives for greater use of evidence that are largely missing from the department’s 
formula grants. A list of federal programs affected by ESSA’s evidence provisions, including 
competitive grants, is in Appendix B. 


=" Technical Assistance: The U.S. Department of Education provides technical assistance to 
states, districts, and schools in two primary ways. IES provides technical assistance to state 
education agencies, districts, postsecondary institutions and other education stakeholders 
through its Regional Educational Laboratories (RELs). The Office of Elementary and Secondary 
Education also provides technical assistance to state education agencies through its 22 
Comprehensive Centers. 109: 110 


The State Role 


Constitutionally, most power over schools originates at the state level.''' While states have delegated 
much of this authority to local school districts, they have also retained substantial influence through a 
combination of accountability systems, monitoring, grants, and technical assistance. The associated legal 
authority and state capacities vary widely from state to state, however.''!? These factors include: 


e State Accountability Systems: Among the many tools that states possess, accountability 
systems probably play the central role in driving change in low-performing Title | schools.'18 
Under ESSA, these systems determine which schools will be subjected to mandatory school 
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William G. Howell, "Results of President Obama’s Race to the Top," Education Next, Fall 2015. Available at: 
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training, coaching or technical support. They also focus on building stakeholder capacity to use research and data to inform 
education programs and policies. The Comprehensive Centers work primarily with state education agency staff, and they focus 
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comprehensive centers is available at: https://www2.ed.gov/programs/newccp/index.html. 

There has been some discussion of combining the RELs and comprehensive centers. See Education Week, "Ed. Dept. Budget 
Could Combine Three Significant Research, Policy Programs," January 25, 2018. Available at: 
http://blogs.edweek.org/edweek/inside-school- 

research/2018/01/regional labs state data_systems_ comprehensive centers in ed budget _2018.html 

Michael A. Gottfried, et al, "Federal and State Roles and Capacity for Improving Schools," RAND, 2011, p. 3. Available at: 
https://www.rand.org/pubs/technical_reports/TR989.html 

Institute of Education Sciences, "State Policies for Intervening in Chronically Low-performing Schools: A 50-state Scan," June 
2016. Available at: https://ies.ed.gov/ncee/edlabs/regions/central/pdf/REL_2016131.pdf;; Institute of Education Sciences, “State 
Capacity to Support School Turnaround,” May 2015. Available at: https://ies.ed.gov/ncee/pubs/20154012/pdf/20 154012. pdf; 
Ashley Jochim and Patrick Murphy, "The Capacity Challenge: What It Takes for State Education Agencies to Support School 
Improvement," Center on Reinventing Public Education, December 2013. https://www.crpe.org/publications/capacity-challenge- 
what-it-takes-state-education-agencies-support-school-improvement; SIRC, “Policy Drivers: Putting the Politics Back in 
Implementation,” January 28, 2018. Available at: http://www.socialinnovationcenter.org/?p=3012 

ESSA’s provisions governing state accountability systems are in § 1111(c). See also GAO, "Every Student Succeeds Act: Early 
Observations on State Changes to Accountability Systems," July 18, 2017. Available at: https://www.gao.gov/products/GAO-17- 
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improvement requirements.''* They determine which schools will be eligible for school 
improvement funding.''® They dictate ongoing performance monitoring.''® They also commonly 
include provisions for stronger interventions for schools that do not improve.''” 


e Technical Assistance with School Needs Assessments: ESSA requires school districts to 
develop improvement plans for each school that has been designated for comprehensive school 
improvement.''® Such plans must include a needs assessment and at least 13 states are 


providing assistance on these assessments.''? These assessments, and the school improvement 


plans in general, are discussed in greater detail in the next section. 


e State-devised Lists of Evidence-based Interventions: Following the needs assessment, 
school districts and schools must develop a comprehensive improvement plan with at least one 
evidence-based intervention that falls within the law’s top three evidence tiers (promising, 


moderate, or strong).'2° Fifteen states are helping schools and school districts by developing lists 


of interventions that meet ESSA’s evidence standards.'2' Such lists may incorporate reviews 
conducted by the WWC at the U.S. Department of Education (discussed earlier). 


e State Approval of School Improvement Plans: At least five states have included provisions in 
their state ESSA plans describing actions they will take to assist schools and districts with their 
school improvement plans, including the inclusion of evidence-related provisions. ‘22 The final 
plans must also be approved by the state education agency, which gives the state additional 
influence over its contents. 128 


e Use of Conditional or Competitive Grants: ESSA allows states to distribute school 
improvement funds on a conditional or competitive basis, which allows them to incentivize the 


choice of interventions with greater evidence. '4 At least 14 states have tied some portion of their 


school improvement funds to the use of such evidence. 


e Ongoing Technical Assistance: Evidence-based programs frequently fail to produce positive 
results because they are poorly implemented.'26 Many ESSA state plans have described how 


they will monitor local implementation, provide technical assistance, and evaluate these efforts. 12” 
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Schools designated for comprehensive school improvement are eligible for Title | school improvement funds under § 1003, 
which replaced the School Improvement Grants program. 

Monitoring requirements are in ESSA, § 1111(d)(1)(B)(vi); See also: Results for America, "ESSA Leverage Points: 50 State 
Report," January 12, 2018, pp. 10-11, 19-21. Available at: https://results4america.org/tools/essa-leverage-points-50-state-report- 


promising-practices-using-evidence-improve-student-outcomes/ 
Results for America, "ESSA Leverage Points: 50 State Report,” January 12, 2018, pp. 34-38. Available at: 


https://results4america.org/tools/essa-leverage-points-50-state-report-promising-practices-using-evidence-improve-student- 


outcomes/ 
ESSA § 1111(d)(1)(B) 


ESSA §§ 1111(d)(1)(B)(iii) and 1114(b)(6); Results for America, "ESSA Leverage Points: 50 State Report," January 12, 2018, pp. 
25-27. Available at: https://results4america.org/tools/essa-leverage-points-50-state-report-promising-practices-using-evidence- 


improve-student-outcomes/ 
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Results for America, "ESSA Leverage Points: 50 State Report," January 12, 2018, pp. 8-9, 39-41. Available at: 
https://results4america.org/tools/essa-leverage-points-50-state-report-promising-practices-using-evidence-improve-student- 


outcomes/ 

ESSA, §1111(d)(1)(B)(v) 

ESSA § 1003(b)(1)(A) 

Results for America, "ESSA Leverage Points: 50 State Report,” January 12, 2018, pp. 15-18. Available at: 
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National Implementation Research Network, "Implementation Drivers." Accessed from http://nirn.fpg.unc.edu/learn- 


implementation/implementation-drivers 
Results for America, "ESSA Leverage Points: 50 State Report,” January 12, 2018, pp. 22-24. Available at: 
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In some cases, they may incentivize schools and districts to work with approved outside providers 
with experience with the selected programs. 


e Continuous Improvement: In their ESSA plans, at least 16 states described continuous 
improvement efforts by schools, school districts, and the state.'28 For example, Tennessee plans 
to study its fastest improving schools and share lessons learned with all of its schools. In 
Vermont, every school and school district must submit continuous improvement plans to the state 
and will receive technical assistance rooted in improvement science.'?2 Georgia has adopted a 
similar Systems for Continuous Improvement framework that also relies on improvement cycles. 


e State-funded Research: Many states are also supporting school improvement through state- 
funded research.'2° Some states have research consortia (often located in large cities) that work 
with schools, including the Tennessee Education Research Alliance, the Education Research 
Alliance of New Orleans, and the Research Alliance for New York City Schools.'3' These 
organizations also receive funding from other sources. When research is conducted or funded by 
the state, it may also be driven by a state-devised research agenda, which can focus limited 
resources on the most pressing research questions.'2 


The Local Role (Schools and School Districts) 


Under ESSA, after a state has designated a school as being in need of comprehensive improvement, the 
appropriate school district must devise a plan for the school that includes one or more interventions that 
are evidence-based. This process can include the following components. '%3 


= Needs Assessments: As was briefly mentioned earlier, the first step after a school has been 
designated for comprehensive improvement is to conduct a formal needs assessment. Such 
assessments commonly review student achievement and identify the causes of the school’s poor 
performance.'** They often review contextual factors, such as the school environment and 
student characteristics, as well as the school’s strengths and weaknesses. '35 They commonly 
draw on school data, focus groups, surveys, and site visits for information. They could also 
include an inventory of existing programs, possibly including a review of their underlying 
evidence. 136 


https://results4america.org/tools/essa-leverage-points-50-state-report-promising-practices-using-evidence-improve-student- 


outcomes/ 
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=" Choice of Evidence-based Interventions: Following the needs assessment, school districts 
and schools must work together with stakeholders to create a school plan that includes at least 
one evidence-based intervention that falls within ESSA’s top three evidence tiers (promising, 
moderate, or strong). '9” 


In an ideal world, this would be a rational process. School improvement plans would focus on the 
gaps identified in the needs assessment, review existing programs, and recommend 
modifications or new programs based on credible evidence. ‘8 In reality, stakeholder politics play 
a central role and these politics may produce plans that are less rational and more influenced by 
local power dynamics. 


Such plans normally involve the formal school leadership, often including the district 
superintendent and/or district staff, school board, and principal of the school.'9° Other 
stakeholders may include teachers, parents, and outside organizations such as teachers’ unions 
or nonprofit organizations that work with the schools.'4° ESSA itself mandates the participation of 
such stakeholders in the development of school improvement plans. ‘41 


Support for any proposed course of action in a school improvement plan may vary substantially 
among the various actors. This support may be influenced by information in the formal needs 
assessment and any evidence reviews, but it is also likely to be shaped by differing political 
beliefs, professional judgment, and the extent to which proposed changes in the status quo are 
viewed as threatening.'4* Such support may also change over time, rising or falling depending 
upon the perceived challenges in the school, perceived success of existing policies, and changes 
in key personnel. '48 


The state can also play a central role in this process. As described earlier, state accountability 
systems can create substantial incentives for local actors to adopt stronger plans. '44 Some states 
may require districts to choose among state-approved lists of interventions and providers. '45 
Under ESSA, the state must also approve the final version of the plan. 146 


The final decision on a school improvement plan is likely to depend greatly on these local and 
state power dynamics and on bargaining among the various actors. These politics will likely 
influence how aggressive, incremental, coherent, or comprehensive the school improvement plan 
will be.'4” The role of evidence in this process will vary. In some cases, it may simply be used to 
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justify a continuation of existing policies. In other cases, it may produce significant change. 148 


= Program Launch: After a school plan has been approved, the next step is to do the early work 
necessary to put it in place. This can include adoption of needed policies, regulations, or 
budgets. '49 It can include hiring, obtaining staff buy-in, training, and contracting with external 
providers. ‘°° It can also include obtaining assets and materials such as work space, practice 
manuals, and data systems for case management, performance monitoring, and audits.'°' 


* Ongoing Implementation: Effective replication of an evidence-based program or practice 
typically requires a balance between fidelity to core program components and adaptation to local 
conditions, combined with sufficient leadership, staff competency and organizational capacity. 15° 


Core components can include practice elements, equipment and related materials, personnel 
qualifications, and training.'®? Fidelity is usually monitored on an ongoing basis to ensure 
performance, sometimes using proprietary metrics or tools. '>4 Fidelity measures vary 
tremendously from model to model, but examples can include class size, task checklists, 
assessments of quality (including through third-party observations), and staff education, training, 
and certifications.'®° 


Such attention to detail is one reason why successful implementation often requires an 
experienced outside intermediary. Higher-capacity schools and program providers may also 
choose to incorporate such fidelity measures into their performance management systems. 15° 


=" Evaluation and Continuous Improvement: After an evidence-based intervention has been 
implemented, there is often room for improvement.'*” Ideally, such innovations should be made 
carefully, since altering a program’s core features departs from fidelity and can produce results 
that are worse, not better. 


Evaluations can answer this question and contribute to the evidence base.'5® Such research can 
be costly, however, and often requires significant outside expertise, often in the form of research- 
practice partnerships. '59 Funding can come from a variety of public or private sources. '6° 
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The Provider Role 


Depending on their choice of evidence-based interventions, some schools may choose to work with 
outside service providers (frequently the model developer) to ensure effective implementation. Many 
model developers have proven track records of successful replication that are validated by rigorous third- 
party evaluations. '® 


Such providers can bring numerous benefits, including materials, validated measurement tools, ongoing 
training, and participation in networks with other schools that are implementing the same program. Some 
providers also update their models to keep them current with the latest developments in the field. '®2 The 
work of one comprehensive school improvement provider, Success for All, is discussed later in this paper. 


Will Evidence-based Reform Succeed? 


Will evidence-based strategies for improving low-performing Title | schools work? At least with respect to 
ESSA, it is too early to know. 


There are several reasons to be pessimistic. One is the modest progress made under NCLB and the SIG 
program, which seemed to show that even dramatic reforms backed by billions of dollars of assistance 
are not sufficient by themselves to consistently improve these schools. 


ESSA‘s evidence provisions are also relatively weak, requiring only that comprehensive school plans 
include one or more interventions that meet the lowest (“promising”) of the law’s three top evidence 
definitions (see Appendix A), with no requirement that such interventions be new to the school.'®° Given 
the central role of stakeholder politics in the plan development process and a likely preference for the 
status quo, there is ample reason to believe that many schools will do little more than the minimum 
necessary, citing whatever evidence justifies their existing activities, and producing little real change. '* 


There are other reasons to be optimistic, however. Decades of previous reform efforts — including the 
standards-based reforms of the 1990s, accountability systems developed under NCLB, targeted 
assistance under SIG, and additional reforms under Race to the Top — have left a legacy and 
infrastructure that remain largely intact.'® This infrastructure failed to produce widespread improvements 
in the past, but this may be because it was tied to a reform agenda that was insufficiently evidence-based. 


Moreover, while some states, districts, and schools may only minimally comply with ESSA’s evidence 
requirements, the history of previous reform efforts under NCLB suggests that some jurisdictions and 
schools will substantially exceed them. Variations in state efforts are already evident in the early ESSA 
plans submitted to the U.S. Department of Education (as described in the section discussing the state 
role). Finally, more jurisdictions may be willing to pursue stronger evidence-based measures because, 


can partner with researchers to apply for a grant from NCER, NCEE, or the EIR/i3 program. 
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unlike NCLB, ESSA does not mandate any particular reform agenda. This could allow schools to adopt 
strategies that are better suited to local conditions and more likely to command local political support. 


While many of these efforts will likely fail, the history of the SIG program suggests that some will succeed. 
Those early successes may be sufficient to spread evidence-based strategies to other states, districts, 
and schools through a process known as policy diffusion.1°° Such diffusion can occur through any (or all 
three) of the evidence pathways described earlier — bottom-up, top-down, and provider-based. 


Many education reform efforts — including curriculum reforms and charter schools — have shown that 
innovations can spread rapidly in this fashion.'®” Even if only some of the early adopters of evidence- 
based practices are successful in the first few years, those early successes — coupled with these diffusion 
processes — may provide the basis for greater adoption of evidence-based programs across the rest of 
the nation. 


Profile: District-Driven Reform 


The San Francisco Unified School District (SFUSD) is one of the few districts in the nation to have 
implemented a successful school transformation initiative under the federal SIG program. The district’s 
work began in 2010 with the establishment of a Superintendent’s Zone, a coordinated strategy for school 
improvement that grew out of an earlier strategic planning effort.'® 


The Zone’s central strategy was based on research by Anthony Bryk and colleagues at the University of 
Chicago Consortium on School Research. Bryk’s work has identified five supports as necessary for 
school improvement, including: (1) building leadership capacity; (2) providing instructional guidance; (3) 
building professional capacity; (4) creating a student-centered learning climate; and (5) strengthening 
parent-community ties. '®9 


The district's Superintendent’s Zone includes 16 of its schools, ten of which were identified as persistently 
low-achieving by the state of California and received $45 million in funding from the SIG program. The 
Zone’s work included many of the district-level activities described earlier in this paper. '”° 


e Needs Assessment: To begin the process, the district created a needs assessment team that 
included the district superintendent, chief academic officer, deputy superintendent, and 
representatives of the district's Research, Planning, and Assessment Department. A data profile 
was created for each of the Zone schools, including information on student demographics, 
student achievement and growth, teacher experience and turnover, and academic and behavioral 
performance and trends, with results disaggregated by race and by program. 


As part of the process, school principals were asked to provide information about their previous 
and current improvement efforts and the supports that they believed were most needed in their 
schools. The needs assessment concluded that “the problem of low performance in schools was 
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Lorraine McDonnell and Stephen Weatherford, Evidence Use and the Common Core State Standards Movement, American 
Journal of Education, November 2013. Available at: http://wtgrantfoundation.org/library/uploads/2015/10/Evidence-Use-and-the- 
Common-Core-State-Standards-Movement.pdf; The number of charter schools in the US grew dramatically after they first 
appeared in the early 1990s, but this growth has recently leveled off. See Robin Lake, "Why is Charter Growth Slowing,” 
Education Next, January 2018. Available at: http://educationnext.org/why-is-charter-school-growth-slowing-lessons-from-bay- 
area/ 

168 This profile draws heavily from AEM Corporation, “Evidence-based Practices in School Improvement: Five Profiles of Promising 
Practices,” November 2016, pp. 54-76. Available at: 
https://www2.ed.gov/about/offices/list/oese/oss/technicalassistance/evidencebasedpracticesschl.pdf 
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a systemic problem, requiring a systemic solution.” 


e Choice of Evidence-based Interventions: The district spent a year studying the Bryk 
framework, including both its research base and implementation requirements. The resulting 
Zone design choices were informed by the framework’s five supports. 


o Building Leadership Capacity: As required by the SIG program, principals were 
removed from SIG-funded schools. The district used strategic recruitment to select new 
principals within the district and/or principals with experience with previous school 
turnaround efforts. Principals were selected who were like-minded, aligned with the 
mission and goals of the Zone, and prepared to lead and implement research-based 
change. 


o Providing Instructional Guidance: Instructional improvement was a central feature of 
the district’s reform efforts, with a heavy focus on implementing a core-curriculum and 
job-embedded, one-on-one coaching.'’' The district partnered with several outside 
organizations to help improve its math and literacy instruction, including Teacher’s 
College, Literacy Collaborative, the WRITE Institute, Partners in School Innovation, 
Algebraic Thinking & the Algebra Project, Project SEED, and Tools for Schools. 


The district also developed one-page documents, called “Salmon Sheets,” that outlined 

frameworks for teaching staff that aligned curriculum, instruction, and assessments. The 
Salmon Sheets included information on expectations, assessments, and details such as 
pacing, scope, and instructional sequences. 


o Building Professional Capacity: The district established Instructional Learning Teams 
(ILTs) at each school to support teacher professional development. The ILTs consisted of 
the principal, an instructional reform facilitator, literacy coaches, two classroom teachers, 
and other key leadership staff. These teams collected information on teacher progress 
through ongoing rounds of classroom observations that relied on a standard protocol. '”2 
This information was then used to make adjustments and inform subsequent teacher 
professional development. 


The district also hired instructional coaches at each school to provide one-on-one support 
to teachers in implementing the Zone’s instructional frameworks. It also created a 
network that connected the school ILTs with one another to share best practices. 


o Creating a Student-Centered Learning Environment: Zone schools adopted an 
extended learning time strategy, including after school and summer learning programs. '”? 
Schools adopted an early warning system to monitor student progress. The schools also 
adopted a college going culture. 


o Strengthening Parent-Community Ties: The district established a family liaison at each 
Zone school and at the overall Zone-level. The schools also adopted a full service 
community schools strategy to connect students and family members with relevant 
community-based partner organizations that provided social services or other assistance. 


171 A review of the literature on coaching and other teacher professional development strategies can be found in: Linda Darling- 
Hammond, Effective Teacher Professional Development (2017). Available at: https://learningpolicyinstitute.org/product/effective- 
teacher-professional-development-report ; 
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173 Min Sun, Emily Penner, and Susanna Loeb, "Resource- and Approach-Driven Multi-Dimensional Change: Three-Year Effects of 
School Improvement Grants," American Educational Research Journal, August 2017. 
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e Program Launch: Launch efforts included recruiting new principals, establishing the Instructional 
Learning Teams at each school, hiring and training instructional reform facilitators and 
instructional coaches, and establishing relationships with external partners. 


e Ongoing Implementation: Continuous improvement was a central feature of the district’s efforts. 
Using a model called the Results-Oriented Cycle of Inquiry (ROCI) developed by Partners in 
School Innovation, the district supported the school-based instructional reform facilitators in their 
continuous improvement efforts. The district also developed a school capacity rubric to rate 
schools based on the Bryk framework. 


e Evaluation: The district established a research-practice partnership between its Research, 
Planning, and Assessments team and Stanford University.!”4 This partnership supports as many 
as 30 ongoing collaborations between researchers at Stanford and the district at any given time. 
This research has helped inform district decision-making and has also provided insights that are 
potentially generalizable outside of San Francisco.'” 


According to an independent study, which used a difference-in-differences methodology to compare the 
district's SIG schools to its non-SIG schools,'”6 San Francisco's efforts appear to have produced positive 
effects on student outcomes.'”’ After three years of implementation, the district’s SIG schools 
experienced improvements in both student attendance and achievement (English language arts and 
mathematics). Families expressed greater satisfaction with the schools. The schools also experienced 
greater retention of effective teachers. 


Profile: State-Driven Reform 


Massachusetts is one of the leading states in the nation on school turnarounds. Unlike other efforts 
across the nation, which have generally been lackluster, the state’s efforts have met with significant 
success.'’8 Why has the state’s experience been so different? One reason may be its singular focus on 
developing and scaling evidence-based practices as a central feature of its school turnaround efforts. 


History and Political Context 


At first glance, the state’s leadership on school turnarounds might seem unexpected. Nationally, 
education reform politics commonly pit centrist Republicans and Democrats against opponents on the left 
(led by teachers’ unions) and on the right (led by social conservatives). Because Massachusetts is one of 
the most liberal states in the nation, with teachers’ unions acting as a major political force, its leadership 
on school turnarounds might seem an exception to the rule.'”9 


Despite its reputation as a bastion of the left, however, the state has been led by a series of socially 
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liberal, business-oriented Republican governors since the early 1990s. '®° The sole exception, Governor 
Deval Patrick, was a business-friendly Democrat who was closely aligned with President Obama. Each of 
these governors supported education policies that were largely aligned with education reformers 
nationally. 


This supportive political environment appeared to produce results. State law governing school 
turnarounds can primarily be traced to two bills enacted by the legislature during this period. The first, the 
Massachusetts Education Reform Act, was enacted in 1993 under Republican Governor William Weld. 18! 
Consistent with the broader push for standards-based education reform that had been occurring 
nationally in the 1990s, the law authorized the development of state curriculum standards and student 
assessments. The state began administering its first student tests under the law in 1998, later expanding 
them after No Child Left Behind was enacted at the federal level in 2002. That same year, the state began 
designating some schools as “underperforming" based on multiple indicators.182 


The next significant expansion came in 2010 under Governor Patrick. Partly to better position the state for 
a federal Race to the Top grant, the administration successfully pushed through the Massachusetts 
Achievement Gap Act.'®° This law substantially strengthened state authority over low-performing schools, 
with the worst being subjected to possible direct state control. The efforts paid off when the state won a 
Race to the Top grant later that year. 164 


Strategies for Scaling Evidence 


School turnaround efforts in Massachusetts, including efforts to scale the use of evidence-based 
practices, largely depend on an infrastructure that was created by these earlier politics and policies. This 
infrastructure — which includes significant state research capacity, accountability mechanisms, and 
targeted assistance — is explained in greater detail below: 


e Ongoing State Research: One strategy that has set Massachusetts apart from other states is its 
ongoing investment in state-funded research, which is overseen by its Office of Planning and 
Research. Shortly after the 2010 law was enacted, the state funded a series of studies that 
evaluated both the effectiveness of its turnaround efforts as a whole as well as the individual 
practices that seemed most associated with schools that were turned around successfully. 


The research — which focused on both implementation and impact — rolled out in stages, starting 
in 2012.'85 In 2014, the state released a study identifying four broad sets of practices that seemed 
most important: (1) leadership; (2) instructional practices; (3) student supports; and (4) school 
climate and culture.'®® Subsequent research, led by AIR, fleshed out subtopics for each of the 
four broad categories, including those related to instruction, expanded learning time, and family 
engagement. '87 
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The underlying research for the practices is maintained on a state web page. '®8 These research 
efforts are also continuing under a state-devised research plan. '89 


e State Accountability System: While research can help identify evidence-based practices, it is 
not usually sufficient on its own to drive local school districts and schools to adopt them. In 
Massachusetts, the central driver in its school turnarounds efforts is the state’s accountability 
system, which was created by the 1993 and 2010 laws, and by subsequent regulations issued by 
the state board of education.'9° This accountability system has also been a major driver of 
increased evidence use in these schools. 


Under the state’s accountability system, schools are placed into one of five categories based on 
their performance.'®' High-performing schools (Level 1) receive substantial autonomy. Mid-tier 
schools (Levels 2 and 3) are subject to increased state oversight and are eligible for targeted 
state assistance. Low-performing schools (Levels 4 and 5) are subject to the greatest regulatory 
oversight, including turnaround requirements and possible state takeover. As many as four 
percent of the state’s schools can fall into the lowest two categories under state law. 192 


Schools that have been designated as Level 4 must create a state-approved school improvement 
plan.'%3 As part of the plan, schools must address each of the state’s four broad sets of evidence- 
based turnaround practices (leadership, instruction, student supports, and school climate), but 
they are also given substantial flexibility on the details, including the selection of individual 
strategies within those sets of practices. '%4 


Despite this flexibility, the accountability system provides substantial incentive for the schools to 
adopt strategies that are effective — the most important of which is the threat of a state takeover. 
The schools are also subject to significant ongoing performance monitoring by the Statewide 
System of Support Office, including annual site visits that assess implementation.'% They can 
track their progress through a state-operated data benchmarking system.'9® They also receive 
additional help in the form of state grants and technical assistance (see below). 


When schools fail to improve they are designated as Level 5 and subject to state takeover. '9” 
Schools designated for takeover are then assigned to a receiver, typically individuals or outside 
nonprofit organizations with a track record of turning around such schools.'98 Four schools have 
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been subjected to this process.'®? Schools must make significant progress before they can exit 
takeover status.20° 


e Grants and Technical Assistance: As they pursue improvement efforts, schools and districts 
are eligible for various forms of assistance. These include School Redesign Grants, which are 
federally-funded competitive grants designed to provide needed resources and capacity. 2% 
Schools and school districts also receive direct technical assistance from the state.2°* This has 
included the development of a field guide based on the state’s earlier research on evidence- 
based practices, a variety of practice-oriented videos, and direct assistance from state employees 
for Level 4 and 5 schools.2% 


Finally, the state has identified what it calls "priority partners" for schools and districts engaged in 
turnaround efforts. These organizational partners have been vetted by the state “through a 
rigorous review process as having a demonstrated record of effectiveness in accelerating school 
improvement.”204 


Overall, the state’s approach to promoting the use of evidence amounts to what some call a combination 
of “pressure and support.” 2° The state school accountability system, including the implied threat of a 
state takeover, provides the “pressure,” although it does so while simultaneously granting schools 
significant flexibility in the choice of individual strategies within the four sets of evidence-based practices. 
The “support” is provided through continued state research, competitive grants, and technical assistance. 


How well has it worked? According to an independent evaluation, the state’s school improvement efforts 
have produced increased student achievement.2°6 Probably as a result, as of January 2017, 57 percent of 
the state’s turnaround schools had exited turnaround status.2°” 


Profile: Provider-Driven Reform 


Model developers are another driver for scaling evidence-based whole school reforms. These 
organizations lack the regulatory and budgetary tools available to states, but they nevertheless play 
central roles in growing and successfully replicating their models, usually through a combination of grant- 
seeking, direct marketing to schools and districts, and ongoing implementation support. 


One example is Success for All (SFA), a reading-focused comprehensive school reform program for 
students from pre-kindergarten through eighth grade. SFA has been shown to generate positive student 
outcomes in several studies in its over three decades of existence.?°8 It is also one of four whole school 
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reform models that has been approved by the U.S. Department of Education as evidence-based.?°9 


SFA’s growth provides insights on provider-based scale. How has the organization expanded the use of 
its model? How does it ensure implementation quality? 


Marketing 


For model developers, scaling comes primarily through direct marketing to schools and school districts. 
While SFA is a nonprofit organization, it must confront the same issues of supply and demand that face 
other products or services. 


On the demand side, the organization has benefitted from large-scale federal policy changes, particularly 
the creation of the Comprehensive School Reform program in 1998 and the Investing in Innovation (i3) 
program in 2010. These programs both provided resources, but the organization still needed to market 
itself to schools to expand its network. Its marketing activities included standard practices such as public 
relations, conference presentations, and the distribution of information through education-related outlets. 
The most effective strategies have usually been word-of-mouth and recruiting new schools through 
existing district partnerships.2"° 


Establishing new partnerships with schools involves more than outreach, however. Cost has also played 
an important role. Effective implementation of evidence-based models requires resources for materials, 
training, and oversight (described below), and such costs must be paid for. In some cases, these costs 
have been subsidized through federal, state, or philanthropic grants. In other cases, however, some or all 
of the costs must be covered by fee-for-service arrangements with the schools, which could amount to 
tens of thousands of dollars per year.2'' A reliance on fee-for-service payment structures has been a 
barrier to scaling SFA and probably for many other providers of evidence-based interventions, as well.2'2 


Once a school has shown interest, but before an agreement is finalized, SFA takes additional steps to 
ensure buy-in from key personnel at the district, principal, and teacher levels. Separate research has 
demonstrated the importance of district support to the success of school improvement programs in 
general.?'$ Principals are also frequently cited an important drivers of school-based reform, second only 
to teachers in importance.?'4 A study of SFA’s work confirmed this finding, showing that its implementation 
quality was substantially associated with principal support.2'5 Finally, SFA also helps ensure teacher buy- 
in by requiring a vote of the staff at all participating schools, with at least 80 percent support required 
before proceeding with a roll out.2'6 
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SFA has grown significantly since it began in Baltimore in 1987. In its first decade, it grew by 5-50 schools 
per year. Following the creation of the federal Comprehensive School Reform (CSR) program in 1998, the 
number of participating schools grew dramatically, peaking at over 1,500 schools by the mid-2000s.?2'” 
When the CSR program ended in 2005, this figure briefly dropped, but it grew again after the organization 
received a $50 million Investing in Innovation (i3) grant in 2010. Today, SFA serves over 1,000 schools in 
48 states. 


Implementation 


Replicating an evidence-based program with fidelity can be a demanding task under any circumstances. 
Such programs commonly come with detailed manuals describing start-up activities and ongoing 
operations. Requirements often include staff training, materials, monitoring protocols, and related 
technologies.?'® 


SFA is no exception. Its core components include curricular materials, training for school leaders and 
staff, and the hiring of on-site school facilitators to oversee implementation.2'9 Such requirements can be 
a challenge under any circumstances, but they are particularly so in persistently low-performing schools, 
which commonly experience demanding teaching environments, budget shortfalls, and high staff turnover. 


“Most schools can’t do this on their own. If they can, they already have,” said Nancy Madden, SFA’s 
cofounder and CEO. “It requires strong, stable leadership and the ongoing support of a well-structured 
intermediary.” 22° 


Continuous improvement is also a central feature of the model. SFA’s participating schools usually 
improve over time.22’ SFA also includes a formal research-backed improvement component. The overall 
model has steadily evolved over the years, with improvements such as the adoption of new classroom 
technologies and professional development strategies. 222 


Strengths and Limitations 


SFA illustrates both the strengths and limits of provider-based scaling of evidence-based programs. 
Strengths include the central role of the model developer as a driver of both implementation quality and 
scale. Providers are well-positioned to market, launch, and oversee the implementation of their models, 
thereby ensuring they are replicated with fidelity (a common challenge in persistently low-performing 
schools). They can also take advantage of network-based economies of scale, including investments in 
research and dissemination of new best practices.228 


However, there are also limits to the strategy. Use of these models depends primarily on provider 
marketing and word of mouth rather than consultation with third party ratings like those of the What Works 
Clearinghouse.?@4 The reliance on voluntary adoption also limits the strategy’s reach to schools that are 
both willing and able to pay to work with an outside vendor. 


that it has not limited the program’s growth or adoption. (Communication with SFA, February 18, 2018). 
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Recommendations 


The history of turnaround efforts in low-performing schools is one marked by failure overall, but with 
pockets of success. Most of these previous efforts were insufficiently rooted in evidence, poorly 
implemented, or both. However, the recent growth of evidence-based whole school reform models, 
coupled with successful efforts like those in San Francisco, Massachusetts, and by Success for All, 
suggest that well-designed and well-implemented strategies can succeed. 


Fulfilling this potential could be made easier, however, if policymakers were to consider the following 
supportive changes: 


e Identify Evidence-based Programs and Practices: Accurately identifying programs and 
practices that are backed by rigorous evidence is a demanding task that requires time and 
expertise that most districts and schools probably do not possess in abundance.?25 


The U.S. Department of Education has provided guidance on ESSA’s evidence definitions for 
formula-funded programs, like Title | school improvement funds.226 However, states will probably 
need to provide further clarity for this guidance to be useful to local school districts and schools. 
At least 15 states have indicated that they will develop lists or menus of evidence-based 
interventions.22” More states should consider providing such guidance. 


As they proceed, however, they should review existing clearinghouse ratings and consult with 
unbiased national experts to ensure that their lists are useful, comprehensive, and appropriate to 
the range of needs identified in school needs assessments.2@8 Such lists should emphasize 
interventions that have demonstrated effectiveness in more than one study, across multiple sites, 
and with relevant populations.22 Such lists should also be flexible and permit school districts to 
choose strategies not on the list if they meet specified evidence standards. 


e Require the Adoption of New or Stronger Evidence-based Interventions that Exceed 
ESSA’s Minimum Requirements: ESSA’s evidence provisions require schools that are identified 
for comprehensive school improvement to have plans that include at least one intervention that is 
evidence-based. However, this requirement does not require schools to do anything that is 
substantially different. It allows schools that are unwilling to change to point to minimal evidence 
justifying their existing work, with little or no alternation to the status quo. 


States or school districts should strengthen ESSA’s minimum requirements by directing plans for 
these schools to include: (1) at least one new evidence-based intervention; (2) at least one 
intervention, new or current, that meets ESSA’s highest (“strong”) evidence standard; and/or (3) a 
proposal to test the effectiveness of at least one new and innovative practice, possibly as part of 
a research-practice partnership. 


e Provide High-quality, Evidence-focused Technical Assistance: Selecting an evidence-based 
program or practice is not sufficient to ensure its effectiveness. Such programs must be 


225 


226 


227 


228 


229 


Patrick Lester, "Report Finds Evidence Clearinghouse Gaps and Challenges," Social Innovation Research Center, “April 11, 
2015. Available at: http://www.socialinnovationcenter.org/?p=1224 


U.S. Department of Education, "Non-Regulatory Guidance: Using Evidence to Strengthen Education Investments," September 
16, 2016. Available at: https://www2.ed.gov/policy/elsec/leq/essa/quidanceuseseinvestment.pdf 

Results for America, "ESSA Leverage Points: 50 State Report," January 12, 2018, p. 28-33. Available at: 
https://results4america.org/tools/essa-leverage-points-50-state-report-promising-practices-using-evidence-improve-student- 


outcomes/ 

Robert Slavin, “We Don’t Do Lists,” January 18, 2018. Available at: https://robertslavinsblog.wordpress.com/2018/01/18/we-dont- 
do-lists/ 

These criteria are relevant to generalizability, sometimes referred to as external validity. In general, an intervention has greater 
generalizability if it is has replicated in multiple settings and multiple studies, preferably with the population of interest. See: 
https://www.socialresearchmethods.net/kb/external.php; See also: Steve Fleischman, Caitlin Scott, and Scott Sargrad, "Better 
Evidence, Better Choices, Better Schools," August 2016, pp. 6-. Available at: https://cdn.americanprogress.org/wp- 
content/uploads/2016/08/30141500/EvidenceESSA-report. pdf 


28 


implemented well and with fidelity to achieve their intended outcomes. Low-performing schools 
will need substantial technical assistance to achieve these results, either from the state, the 
federal REL program or Comprehensive Centers, or providers. While such assistance will likely 
be provided by a variety of sources, quality will matter. Massachusetts provides an example of the 
targeted use of evidence-focused assistance, including its use of field guides, practice videos, 
direct technical assistance, and grants. 


e Encourage the Use of Providers or Other Intermediaries Experienced with Evidence-based 
Programs: In many cases, state-provided technical assistance may not be sufficient to ensure 
effective implementation. Model developers and other external intermediaries are often critical to 
ensuring model fidelity in low-capacity, low-performing schools that frequently face budget 
limitations, challenging teaching environments, and high staff turnover. 


Providers usually have deep experience with their models, including fidelity and adaptation 
issues, and are typically well equipped to provide the training, materials, technology, and 
monitoring protocols necessary for effective implementation. Such providers also commonly 
maintain networks of schools implementing the same model who can consult with one another 
through communities of practice. 


The Success for All example provides insights on the work of one whole school reform provider 
with evidence of success. San Francisco worked with numerous outside providers as part of its 
district-driven work. Massachusetts identified “priority partners” for its turnaround work, including 
experienced individuals and organizations to oversee state takeovers of schools in situations 
where turnaround efforts have failed. 


In addition to identifying lists of evidence-based interventions, as described above, more states 
and school districts should follow the lead of Massachusetts by vetting and approving providers, 
consultants, or other intermediaries based on their track records with evidence-based change. 
States and districts should also ensure that schools have the necessary authority and funding to 
work with these providers. 


e Integrate Evidence into State Accountability Systems: The Massachusetts example 
demonstrates the value of tying evidence to existing accountability systems. Massachusetts 
schools that have been identified as turnarounds must select from a state-devised menu of 
strategies and evidence-based practices. Those that do not improve are subject to state takeover 
by designated organizations with a successful turnaround track record. Such accountability 
provisions may provide greater leverage for evidence-based change. 


In their ESSA plans, at least 16 states described plans for more aggressively intervening in 
persistently low-performing schools using strategies that leveraged needs assessments, 
continuous improvement strategies, and/or evidence-based interventions.*2° More states should 
consider integrating evidence into their accountability systems. 


e Incorporate Evidence More Thoroughly into State Grants: Evidence has been a substantial 
component of federal competitive grants for several years.2°" In their ESSA plans, at least 14 
states indicated that they will also tie school improvement funds to the use of such evidence.?°2 
More states should incorporate evidence into their grants to districts and schools, either as a 
grant condition or as a component of a competitive grant. 
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After decades of failure, ESSA’s evidence provisions provide an opportunity to bring evidence-based 
change to the nation’s lowest-performing schools. The success of such efforts is not assured, however. 
There will be challenges even for those schools that are committed to reform. 


To succeed, ESSA’s framework must be viewed as a floor, not a ceiling. The opportunity offered by ESSA 
will become a reality only if federal, state, local, and provider efforts are all fully utilized and aligned. 


—-o-— 
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Appendix A: ESSA’s Definition of Evidence-Based 


ESSA‘s definition of “evidence-based,” as used throughout the law, is an “activity, strategy, or 
intervention” that: 


(i) demonstrates a statistically significant effect on improving student outcomes or other relevant 
outcomes based on— 


(I) strong evidence from at least 1 well-designed and well-implemented experimental 
study; 


(Il) moderate evidence from at least 1 well-designed and well-implemented quasi- 
experimental study; or 


(Ill) promising evidence from at least 1 well-designed and well-implemented 
correlational study with statistical controls for selection bias; or 


(ii) 
(1) demonstrates a rationale based on high-quality research findings or positive 
evaluation that such activity, strategy, or intervention is likely to improve student outcomes 
or other relevant outcomes; and 


(Il) includes ongoing efforts to examine the effects of such activity, strategy, or 


intervention. 


Source: ESSA, § 8101(21)(A). These definitions are further defined in U.S. Department of Education, 
"Non-Regulatory Guidance: Using Evidence to Strengthen Education Investments," September 16, 
2016. Available at: https://www2.ed.gov/policy/elsec/leg/essa/quidanceuseseinvestment.pdf 


31 


Appendix B: Programs Affected by ESSA’s Evidence Provisions 
Sections of ESSA that are affected by its evidence definition include the following: 


Title | - Programs for the Disadvantaged 


School Improvement (Secs. 1003, 1003A, 1111(d), 1114, 1115) * 
Parent and Family Engagement (Sec. 1116) 
Prevention for Children/Youth Neglected Delinquent & At-Risk (Sec. 1414) 


Title Il - Teachers and Principals 


Formula Grants (Secs. 2002, 2101, 2103) 

Teacher and School Leader Incentive Fund Grants (Sec. 2212) * 

Literacy Education for All, Results for the Nation (LEARN) (Secs. 2221-2225) * “ 
American History and Civics Education (Sec. 2233) * 

Supporting Effective Educator Development (SEED) (Sec. 2242) * 4 

School Leader Recruitment and Support (Sec. 2243) “ 


Title IV — 215t Century Schools 


Student Support and Academic Enrichment Grants (Secs. 4104, 4108) 
21st Century Community Learning Centers (Secs. 4204, 4205) “ 
Magnet Schools Assistance (Sec. 4406) “ 

Family Engagement in Education Program (Sec. 4503) * “ 

Education Innovation and Research (Sec. 4503) “ 

Promise Neighborhoods (Sec. 4624) * “ 

Full Service Community Schools (Sec. 4625) * 4 

Academic Enrichment (Sec. 4644) * 


Title VI —- Native American and Hawaiian Education 
Improvement of Educational Opportunities for Indian Children and Youth (Sec. 6121) 


Notes 


* 


Requires or incentivizes the top three tiers of evidence (strong, moderate, promising). 
* Competitive grant. 
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